With the proliferation of digital cameras in portable, ruggedized, and desktop devices, there is a related desire to use these cameras in new ways. Some of these ways work with faces that are within the camera's field of view. By detecting and interpreting faces, many new functions are provided, such as identifying a known user, interpreting a user's expressions as commands or as other input, mapping a user's expressions to an avatar for a chat session, determining whether a user is paying attention to something in the user's field of view, and more.
Facial expression is the most powerful nonverbal means for human beings to regulate communication with each other and to regulate interaction with the environment. With a facial expression recognition system different categories of facial expressions can be automatically identified in static images or in dynamic videos. The facial expression category identification can be used in many applications such as intelligent human-computer interaction (HCI), social robots, gaming and entertainment consoles and systems, online education and marketing services, medical treatment and diagnosis, etc.
In many techniques, in order to recognize and interpret faces, first a face region is identified. This may be done for a single image or for a sequence of images in a video. The images may be two or three-dimensional. Once a system determines that a particular region may have a face, then the alignment of the face is determined. Facial landmark points may then be used to determine the direction toward which the head is facing, to track eye movements, to recognize or distinguish two different faces and more.
After a face has been detected, the face can be analyzed for a facial expression. The appearance of a face may be analyzed using local representative features such as LBP (Local Binary Patterns) and Gabor features to describe the appearances of samples of different facial expressions. This kind of method is well suited for high volume processing in which there are many facial expressions to analyze. Alternatively, facial muscle movements may be analyzed in terms of the displacements of a set of unique action units (AUs). Traditionally, there are 46 manually defined facial constituent components referred to as AUs. This large number of components allows for small variations and complex facial expression variations to be recognized.